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A method of adjusting for bias due to nonresponse to 
an alumni survey is demonstrated, based upon analysis of data from 
both on-time and late, respondents. The responses are opinions of 254 
alumni regarding how they rate their recently completed program of 
study within a college of an urban university. As part of the 
university's curriculum review process, a questionnaire was developed 
to obtain alumni ratings of thei«r degree program. Twenty-two 
questionnaire items were selected from the University of Illinois 1 
Program Evaluation Survey (Smock and Hake, 1977), and five 
demographic questions and a final "would ycu recommend the program?" 
question were added. Subjects had received an undergraduate or 
graduate degree during the last 5 years. It was found that on-time 
and late respondents differed significantly in their attitudes toward 
specific aspects of the academic program. Adjustment's were made in 
the average ratings to correct, for these differences. In addition, 
after allowing for random sampling errors, the unadjusted results 
were contrasted with the adjusted ones. The findings demonstrate that 
for certain responses, the adjustments were relatively minor , while 
tor other questions the changes were quite significant. In. eight 
cases, the adjusted ratings were ' significantly less critical of the 
program than were the unadjusted ratings, based only on the responses 
. of on-time respondents. It is suggested that the increasing reliance 
on alumni responses to mail questionnaires and their use in the 
internal review and decision-making process of universities make 
proper analysis of the data crucial to institutional researchers. The 
literature on survey nonresponse is also reviewed. (SW) 
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, Abstract - 

The increasing involvement of alumni and other external 
members of the university community in program review has resulted 
in a. reliance on survey instruments t<p obtain their evaluations. 
Decision-makers in higher education have found that mail question- 
naires are a convenient vehicle for gathering this information. 
However, the typical low response rates . of such surveys have focused 
researchers 1 attention on procedures for reducing potential non- 
response bias. This paper demonstrates a method of adjusting for 

bias due to nonresponse, based upon> an analysis of data^sfrom both 
J. 

op-vtime 'and late respondents. The ^responses are opinions of 254 
alumni regarding how they rate their' recently completed program of 
study within a college of an urban university. 
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Introduction 

Alumni ratings are being used increasingly as a source of 

m evaluative information 'in universities. According to Wis£, Hengstler 

and Braskamp (1981), the greatest potential utility of alumni evalua- 

tions appears to be for purposes of program review. Clark, Hastnett 

and Baird (1976) argue that recent alunrii have a better perspective 

about the procedures, requirements and contents of a program' than do, 

students- and that they .tend to be more objective than faculty mem- 

^bers. Based^on a survef of department chairmen ''at 134 institutions, 

/ 

Clark (1977) ^reported that almost 60 percent of the university admin- 1 

istrators sampled considered alumni opinions to be very important* 

, / 

information in university .review and evaluation procedures. , - 

Reliance on input from alumni as part of the program review 
process necessitates the use of surve^ to obtain the opinions sought. 
A convenient, low co n st method which is utilized by many universities 
is the mail questionnaire. For example, the University of Illinois 
mails an alumnji. survey to all degree recipients one year after gradua- 
tion as a routing practice. As yet, however, the data from these 
surveys have. not been incorporated ^Lnto the regular review process. * 
Before endorsing the use of alumni ratings, Wise et al (1981) raised " 4 
severair^is^sues dealing-^^th potential sources of measurement error. 
They addressed these measurement issues to determine what influence, 
if any, they might have on alumni ratings . Their study demonstrated 
that ^lumni ratings do, in fact, provide a relevant and unique source 
of data which^is desirable to include in the assessment of the quality 
of a department 1 s program. 

Their use of ' questionnaire items from the standard Program 

• ~ * % 



Evaluation Survey (PES) , plus the results from' their research 
investigation, allay many expressed concerns about the internal 
validity* of alumni ratings. Nonetheless, they did not address other 




types of potential/errors associated with the use of alumni opinions 
in the program review process. Of particular concern are external 
validity and reliability of responses to a mail' survey of alumni. 0 

A Research Perspective 

External validity and reliability of research results are very 
common, but also very broad, issues typically raised regarding sUr- 
vey research. Examining the potential sources of these errbrs al- 
lows a researcher either to minimize errors due to specific sources 
through careful research design, or to adjust for biased results 
after the fact. Frame and selection sources of external validity 
errors, can be minimized through the use of a sample frame which is 
representative of the alumni population and a selection procedure 
which allows each alumnus an equal chance of beifig chosen to com- 
plete the survey instrument^ Errors due to norfresponse , however, 



'are not as easily eliminated through research design efforts. When 
the percentage of nonrespondents is high «for a given survey , the 
prospect for significant systematic bias in the respondent data 

should be investigated. Also, a low response rate deduces the 

* * ^ 
N effective sample size for the study a,Qd necessarily decreases thja 

* reliability* of sample estimates generated from thenctata . Conse- 

quently, preliminary and concurrent efforts to increase the response 1 
f 

rate to a survey enhance both the external validity and the relia- 
bility of the results,. Techniques employed to correct for non- 
response bias, on the o 4 ther hand , mainly improve the external validity 
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of the research. 

The problem of nonresponse to surveys has haunted survey re- 

) 

.searchers in all fields for years. Advance notification to sample 
members and techniques employed Regarding the questionnaire itself, 
the cover letter, the 'envelope and many other components of the sur 
vey may^be marginally successful in increasing 'the response rate. 



Still, very often, a large percentage of the intended sample has 
not responded by the deadline. After th£f fact, 'the simpliest ap,- 
proach for dealing with potential nonresponse bias is to assume 
that nonrespondents have the same key characteristics £s those., re- 
sponding to the survey. -Extrapolating the results of the data 
ai>alys:Ls to the nonrespondents a'ssumes equality of the two groups,- 
which is simply a matter of blind faith and is not a Recommended 

A 

procedure. t j 

A second tact is to compare known demographics -of the nonre- 
spondents to those, who respond with the intent of showing that re- 
spondents and nonrespondents do not differ significantly based on 
age, sex, socio-economic status and other demographic variables. 
Comparing demographics is an improvement over th£ "blind faith 11 
methoct, but important attributes being investigated in the survey 
may be independent! of the available demographics.'^' 

A more burdensome procedure for measuring nonresponse bias 
utildzes followup interviews, usually via the telephone, to deter- 

/ 

mune the extent of the differences, if any, between respondents and 
nonrespondents. The followup procedure is the most desirable 'of . 
the three methods mentioned, . but it is also the most expensive and 
time consuming. • »In addition, by altering the medium of questioning 



the followup method 'raises additional issues of possible interviewer 
" bias and other measurement differences between those responding to 
mail questionnaires versus telephone calls. Consequently, followup 
attempts to improve the external validity of a survey may result in, 
the internal validity of the data gathering being questioned. 

* ' ■ ' J 

Purpose _ . # . . • 

Since each of the above means of dealing with survey, nonre- 
sponse has major drawbacksy the purpose of this paper is, to of£er a 
practical alternative to the methods -previously discussed for* adjust- 
ing for nonresponse bias. An estimate, of nonresponse bias and a ^ro- 
cedur'e for correcting the resultant systematic* errors-'in an alumni 
survey are based on an analysis Qf on-time and late respondent data'. 

Reducing and Correcting Nonresponse Bias ' * 

4 t 

r ** 

The literature of the social sciences an^ related disciplines 

has dealt with the problems associated w^itlj. survey nonresponse in 

a variety of wpfys. The greatest amount of attention-. has been devoted 

• ». . 

to method^ of stimulating responses to mailed questionnaires (Linsky, 

1975) . Methods to increase the response rate have been classified 

according to the timing of the efforts and the technique used. 

Research evidence indicates that advance notification by telephone 

is effective in increasing response rates and^^Lao^acceler'ates the 

fate of return., However, reminders, especially successive followups, 

though costly, appear to be a better investment .than preliminary 

notification (Kanuk and Berenson, 1975) . . ' r 

* Other efforts to stimulate the response ^ate include a variety 

of techniques classified as concurrent methods, since they are in- 

corporated in the, major mailing of the questionnaire. In their 
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review of studies which have focused on the effectiveness of these 
, techniques, Kanuk' and Berenson (19 75) report* that the results re- 
garding questionnaire length, personalization of the/ mailing > in- 
fluence of the cover letter, promise of anonymity, questionnaire 
size, method of reproduction, and color of the questionnaire are 
inconclusive about the effect on response rate. Their examination 
of research findings, however, does indicate that of ficial \>r re- 
spected Sponsorship, stamped return envelopes, special delivery and 
air mail 'postage, and a twenty-five cent incentive are effective in 
increasing response rates.. The results of the studies also showed 
that deadline dates did not increase the response rate, but did 
accelerate the speed of questionnaire return, A more recently com- 
pleted study by Hornik (1981) showed that both response rate and 
response speed can be Enhanced', without causing response bias, by 
indicating ija the 'cover letter ' that only • a few minutes of the re- 
spondent's time was needed to complete the questionnaire. 

Another body - of "knowledge concerning , survey nonresponse focuses 
on how to analyze data already collected so that* investigators can 
•account for or correct nonresponse bias. A number of researchers 
^ have attempted to identify .the salient differences between respon- 
dents and nonresponctents so that the degree of bias can be estimated 
and a correction determined which would make the results of a survey 
more representative of the sample universe, • Thespe efforts have 
focused on "demographic, socio-economic and personality variables. 

The only consistent finding is that respondents tend to be 
better 'educated than nonrespondents (Kanuk and Berenson, 1975f. 
Wallace (1954) reported virtually no difference between respondents 
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and nonrespondents in occupation and a number of other socio-economic 
characteristics; while Robins (1963) discovered higher level occupa- 
tions among respondents, but no significant differences in social or * 
personality variables, Ognibene (1970), on the other hand/ found 
respondents to be higher in leadership, gregar iousness , and reading 
habits, \ ^ 

Besides educational differences, Donald (1960) suggests that the 
most promising approach Jor classifying respondents and nonrespondents 
may be to assess the interest or involvements of respondents versus 
nonrespondents in a particular investigation. Her recommendation is 
based on: (1) evidence that response rates tend to be high when re- 
spondents have a special interest or involvement in the subject being 
studied (Ferness, I960), (2) indications that high levels of interest 
or involvement are more characteristic of early respondents (Cartwright, 
1949), and (2) her own research findings that late respondents are 
less likely to place a high value on the. organization being evaluated 
(Donald, 1960) . Other examinations of early versus late response 
bias have found no demographic differences, other than in employment 
and occupational variables (Newman, 1962; Shuttleworth, 1940) . 

The correlation between speed of response and involvement in 
the organization has led to efforts to extrapolate trends in Responses 
£o estimate nonresponse bias. The basic assumption of such efforts 
is that respondents who answer later are more like those who do not 
respond at all than those who answer sooner. ' if researchers assume 
that the last wave or combination of the later waves are representa- 
tive of all the nonrespondents, then they can justify weighting the 
nonresponses by the late replies. Sometimes it is possible to 

10 
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establish -trends from results of several response waves and then to 
weight the nonresponse by continuing these trends. However, there 
is danger in assuming a linear trend. The curve produced may in- 
dicate a point beyond which more of the same degree of unfavorable 
or uninvo^ved members of the sample will return questionnaires 
(Donald, 1960) . Another problem is that there may be a point at 
which the direction of the. curve changes radically. For example, 
Baur (1947) discovered that replies from the. slowest respondents 
more closely resembled the earliest respondents than they did the 
intermediate respondents. If such 3. U-shaped distribution is not 
evident, then a simplier mode of analysis than determining a trend 
is to assume that had nonrespondents given a response, it would have 
been unlikely to fall at the most favorable end of an attitude scale. 
Rather, than analyzing a sample of nonrespondents and weighting all 
the nonrespondents. according to the results of this analysis, as 
proposed by Dalenius (1961) , it appears reasonable to substitute 
late respondents for nonrespondents in the weighting formula to 

f 

derive population estimates. 

Research Methodology 

The data source for this study consists of responses to a survey 
of alumni of a single college of a university located in a large, 
midwestern American city. As part of the University's curriculum 
review process, a questionnaire was developed to obtain ratings from 
alumni regarding opinions of their recently completed degree program. 
To minimize internal validity problems associated with measurement 
sources of error in the development of the evaluation instrument, 
twenty-two relevant questionnaire items' were selected from the 
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University of Illinois' Program Evaluation Survey (PES) (Smock and 
Hake, 1977) . in addition, five demographic questions and a final 
"would you recommend the program?" question were added to complete 
the instru/nent . » 

A list of alumni who had received an undergraduate or graduate 
degree during the previous five years from the college being evaluated 
served as the sample frame for the study. The committee conducting 
the evaluation and university officials believed that less recent 
alumni would have difficulty recalling certain dimensions of the pro- 
gram under evaluation. Also, curriculum changes six years previously 
limited the relevant graduation period to five years. The alumni 
list was culled to eliminate duplicate names, i.e. holders of two 
degrees. In addition, alumni were instructed to respond to the ques- 
tions as they related to their most recently completed degree program 
in the college. A total of 1,749 alumni were selected for the study, 
each with an equal chance of being chosen from the frame. The maxi- 
mum sampling error of a sample proportion at the 95% confidence level 
for a sample of this size is only 2.3 percentage points. 

Meticulous questionnaire design, in combination with the use of 
questions from the PES questionnaire, and exacting sample frame 
definition and selection procedures educed the potential for measure- 
ment, frame and selection error sources in the alumni survey. 
Sampling error also would be minimized if the size of the resultant 
sample approaches the planned sample size. Consequently, the major 
threat, to reducing the validity and reliability of the survey results 
was a low response rate. 

Encorporating the results from previous .research regarding con- 
current techniques which stimulate the response rate, a- cover letter 
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on university stationary was written by the chairperson of the univer- 
sity's curriculum review committee to authenticate the study. Alumni 
were advised of the importance of their opinions in the review pro- 
cess, promised that the exercise would take only a "few moments" of 
their time, guaranteed anonymity, and provided with a stamped return 
envelope. in addition, since a deadline has been found to accelerate 
the rate of questionnaire return, it was requested in the cover letter 
that alumni return the completed questionnaire within ten days. Using 
similar procedures, including a nonper sonalized cover letter and no 
followup postcard, Cox, Anderson and Flucher (1974) attained a 13.2% 
response rate to their evaluative mail questionnaire by the end of a 
16-day period. 

Findings 

By the deadline date, only 120 of the 1,749 questionnaires had 
been returned to the institutional research office at the university. 
With the hopes of increasing the response rate from 6.9% to at least 
the 13.2% achieved by Cox et al (1974), the cutoff date for analyzing 
the data was postponed one week to allow returned questionnaires to 
find their way through the university mail system. By weeks end, an 
additional 33. questionnaires had been received, bringing the total 
"on-time" rejsponse rate to a dismal 8.6%. At that point, time con- 
straints forced the committee to analyze the 151 responses in hand 
and to report its findings. Average ratings on the twenty-two seman- 
tic differential items, as reported for the 151 on-time respondents , 

I 

are shown in the first column of Table 1. ' 

- - INSERT TABLE 1 HERE - - 



Over the next month, 103 "late" questionnaires were received by the 
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institutional research office. The late response rate of 5,9%, based 
on the total questionnaires mailed, increased the overall rate of 
response to the alumni survey to a somewhat more respectable and 
expected l^vel of 14.5%, However, the late questionnaire data were 
not analyzed by the curriculum review committee and consequently 
never were included in the committee report. Although there were no 
apparent trends in the responses on the later questionnaires, it did 
appear from a cursory inspection that late respondents were less 
critical of the program under evaluation. 

An analysis of the late responses verified that, in fact, there 
were no trends in the data. Since it was impossible to establish any 
trends, the late responses were grouped as representative of opinions 
of nonrespondents, had they returned their questionnaires. Average 
ratings on the twenty-two items for the 103 late respondents appear 
in the second column of Table 1. Comparing these average ratings with 
those for the on-time respondents revealed a very consistent, system- 
atic bias. For all but the first three evaluative criteria, the late 
respondents were either equally critical, or in most cases less 
critical, of the program than the on-time respondents. (For the first 
three items, a neutral rating, rather than one skewed toward either 
pole, is probably the most desirous value) . This bias is borne out 
by the additional finding that 87.5% of late respondents, compared 
to only 80% of on-time respondents, indicated that they would recom- 
mend their program of study to interested students. 

The results of difference of means tests between on-time and 
late respondents' scores show in" Table l'that late respondents were 
significantly les's critical than on-time respondents with respect to 
the following evaluative criteria: (1) the level of the program, 
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(2) the texts and instructional materials, (3) the quality .of 
instruction., (4) student/faculty contact, (5) the value of the pro- 
gram, (6) student attitudes toward the program, (7) faculty attitudes 
toward the program, and (8) overall satisfaction with the program. 
Six of these eight differences are statistically significant at less 
than the .05 significance level (third column of Table 1), indicating 
a much less favorable response from on-time respondents. The direc- 
tion of this response bias is opposite that found by Donald (1960) . 

Assuming that late respondents* are more representative of non- 
respondents than are on-time respondents, the average ratings from 
late respondents were weighted by the large nonrespondent sample size 
to derive an adjusted average rating for each item. The ratings, 
adjusted for nonres^ponse bias, appear in the fourth column of Table 1. 
The adjusted averag^t ratings were then compared to thg 95% confidence 
interval limits associated with the on-time sample 'means (shown in 
the last column of Table 1) . It should be noted, that in all eight 
cases where the on-time and* late respondent average ratings are signif- 
icantly different, the adjusted ratings fall outside the confidence 
limits. For the other 14 nonsignificant items, the adjusted average 
ratings are- within the confidence limits. In other words, the system- 
atic errors associated with the response bias to eight of the 22 
items were greater than the self -compensating, random sampling errors 
^allowed for by a 5% significance level. These § findings are consistent 
with the observation made by Ferber (1948) that: M The problem of 
response bias must be considered with specific reference to a par- 
ticular question or characteristic. The presence of bias in one 
question does not mean a priori that the replies to other questions 
on the same questionnaire are also biased . M 
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In an attempt to determine if on-time respondents could be 
distinguished from late respondents, an investigation of demographic 
and socio-economic differences between the^two groups was undertaken 
Alumni responses to questions about: (1) the university degree 
received and the department in which they majored, (2) their occupa- 
tional status, and (3) how closely their job related to their major 
course of study were not significantly related to the speed of their 
responses. As was found in most studies reviewed by Kanuk and 
Berenson (1975) , the data collected in this study provided no basis 
to conclude that there are discernible demographic or socio-economic 
differences between on-time and late respondents. 

Discussion 

The results reported in this paper show that, at least in the 
case of the one alumni survey described, on-time and late respondents 
do differ significantly from each other in their attitudes toward 
specific aspects of the academic program they recently completed. 
Adjustments are made in the average ratings to correct for these 
differences; then, allowing for random sampling errors, the unadjusted 
results are contrasted with the adjusted ones. The findings demon- 
strate that for certain responses, the adjustments are relatively 

minor, while for other questions the changes are quite significant. 

f 

In eight cases, the adjusted ratings are significantly less critical 

of the program than were the unadjusted ratings, .based only on the 

■1 

responses of on-time respondents. ' ' 

This paper demonstrates to practitioners the why and how of a 
method which can be used to adjust for nonresponse bias in survey 
data. The increasing reliance on alumni responses to mail question- 
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naires and their use in the internal review and decision-making 
process of universities make proper analysis of the data crucial to 
institutional researchers. For theoreticians, this study provides 
added evidence which confirms the usefulness of utilizing strength 
of attitudes and feelings toward a ^subject when correcting for non- 
response bias in survey data. 
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TABLE 1 



Alumni-^Program Ratings and 



Respondent 
Ratings* 



On-time Late 

Semantic Differential Items ' N=151 N=103 

^1. Framework of program N 
(theoretical/practical) 

2. Amount of structure in curriculum 

(flexible/rigid) 

3. Orientation of course work ' 

( det ai 1 ed/gen er al) 

4. Variety of course offerings * 

(few/many) 

5. Required courses (integrated/ 

nnre]/at§d) ' 

6. Level of program (easy/difficult) 

7. Program's match to student abilities 

(challepging/not challenging) 

8. Texts and instructional materials 
(poor/good) . 

Quality of instruction (high/lown 
Evaluation procedures (unfair/fair) 
Enrichment activities (available/ 

unavailable) 
Student/Faculty contact (impossible/ 
possible) 

Faculty orientation to student needs 

(concerned/indifferent) 
Quality of academic advising 

(low/high) 
Providing credentials for employment 

(emphasi zed/ignored ) v 
Quality of vocational counseling 

(low/high) 
Attention to programs \ 

(undergraduate/graduate emphasis) • 



2.99 


2.85 


3.15 


3.17 


3.03 


2". 86 


2.60 


2.62 


2.43 
3.21 


2.39 
3 .|44 

J 


2.48 


2 j 3 5 


3.19 ' 
2.66 

J.DO 


3.52 
2.45 

■J • / O 


3.48 


3.49 


3.62 


3. '83 


2.53 


2.'38 


2.81 


2. 80 


3.14 


2.95 


2.35 


2.42 


2.41 


2.51 
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ificance Tests 



E»if f erence 
of Means Test 
Significance 
Levels frO . 

.87 

.16 

.90 

.74 
.03 a 

.30 

.01 a 
.09 a 
.67 

.99 

.09 a 

.26 

' .99 

.21 

.66 

a 

.48 



Ratings 
Adjusted 
for " - 
Nonresponse 

2.87 
3-. 1.6 
2.88 
2.62 



2.39 
3.42 



2.36 

3.49 
2.47 
3.72 

3.49 

3.82 

2.40 

2. 80 

2.96 

2.41 

2.50 



, 95% 
Confidence 
Interval 
Limit 

2.83 

3.30 

, 2.88 

- 2.77 

2.27 
3.35 

■-. '2.33 

3.35 
2.49 
3.8 2 

3.68 

3.78 

2.36 

2. '61 

2.96 

2.52 

2.57 
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21. 
22. 



TABLE 1 ; 

Alumni Program Ratings and Significance Tests 

/ / 



S 



Semantic Differential Items 

Value of program (worthwhile/ 

useless) *" 
Student attitudes (-indifferent/ 

dedicated) 
Faculty attitudes (dedicated/ 

indifferent) 
Overall, satisfaction (very/ 

not at all) 
Preparation for professional life 

(not/very helpful) 



Respondent 
Ratings 

Late 
N=103 



On-time 
N=151 



2.08 
3.30 
2.61 
2.46 
3.46 



1.84 
3. '65 
2.27 
2.20 
3.64 



Difference 
of Means Test 
Significance 
. Levels PC • 

.04 a 

.003 a 

.01 a ' 

.03 a 

.24 



' Ratings . 
Adjusted . 
for 

Nonresponse 

1.86 
3.62 ( 
2.30 
' 2.22 
3.62 



95% 
Confidence 
Interval 
Limit 
t . . 

1.93 
3.45* 
2.45 
2.30 " 
3.64 



a \ . ■ » 

Interpreted as significantly different ratings. u 
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